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Abstract Body 
Background and Purpose 


Despite increasing consensus that social-emotional learning, or SEL, is an important 
contributor to success in school and life, few tools are available for educators to assess SEL skill 
and to use assessment results to inform educational practice. In particular, there are no scalable, 
feasible, usable, and scientifically sound assessment systems that measure social-emotional 
comprehension, which includes mental processes enlisted to encode, interpret and reason about 
social and emotional information. Social-emotional comprehension includes the abilities to infer 
others emotions from nonverbal cues, to take others’ perspectives, to solve social problems, and 
to enlist cognitive strategies involved in self-control. Well developed social-emotional 
comprehension is associated with success in school and life (McKown, Allen, Russo-Ponsaran, 
& Johnson, 2013; McKown, Russo-Ponsaran, Johnson, Russo, & Allen, 2015; McKown, Russo- 
Ponsaran, Allen, Johnson, & Warren-Khot, 2015). 


To address this need, with support from the Institute of Education Sciences, we 
developed SELweb, a web-based assessment system for students in Kindergarten through third 
grade that is designed to assess four dimensions of social-emotional comprehension (Lipton and 
Nowicki, 2009), including emotion recognition or “Social Awareness” (Nowicki & Duke, 1994), 
social perspective-taking or “Social Meaning” (Happé, 1994; Wellman & Liu, 2004). Social 
problem-solving or “Social Reasoning” (Bauminger, Edelsztein, & Morash, 2005; Crick & 
Dodge, 1994), and “Self-Control,” or mental processes involved in delaying gratification and 
controlling emotions to achieve a goal (Duckworth, 2011). 


We present a summary of evidence from three studies of the psychometric properties of 
SELweb. In addition, we provide an interactive electronic demonstration of SELweb describing 
features that facilitate its broad use in education settings. 


Methods 


Setting and participants. This research took place during three academic school years 
and includes an ethnically and socioeconomically diverse sample of 8,881 children from 
kindergarten through third grades. 


Assessment description. SELweb is a web-based system to assess social-emotional 
comprehension in kindergarten through third grade, designed to maximize usability in schools. 
Throughout the research project, education partners’ input was sought and integrated into 
assessment procedures, and the design of assessment report output. From the first field trial, 
assessment results were shared with education partners to use to understand their students and 
programs. 


For the social awareness module, children viewed digitized photographs of faces and 
clicked to indicate whether the face reflected happy, sad, angry, scared, or just okay. Item 
scoring for all SELweb modules is described in Table 1. 


For the social meaning module, we created 12 illustrated and narrated vignettes in which 
a character is disappointed, scared, sarcastic, lying, hiding feelings, or harboring a false belief. 
After each story, children were asked a question whose correct answer required accurate 
inferences about the story character’s mental state. 


For the social reasoning module, we created five illustrated and narrated vignettes 
involving ambiguous provocation and five involving peer entry. After each vignette, children 
selected: (a) a description of the problem, (b) a social goal, and (c) solution preference. To 
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reduce respondent fatigue, we created five test forms with six vignettes each. Each form included 
three ambiguous provocation vignettes and three peer entry vignettes. Each vignette was 
included on three forms. 


To assess self-control, we developed a choice-delay task (Kuntsi, Stevenson, Oosterlaan, 
& Sonuga-Barke, 2001) and a frustration-tolerance task (Bitsakou, Antrop, Wiersema, & 
Sonuga-Barke, 2006). 


Research Design 

This research included three psychometric field trials. In two field trials SELweb and 
validation measures were concurrently administered. In all field trials, a subset of children 
completed SELweb twice during the school year, permitting an estimate of temporal stability. 


All students in kindergarten through third grade completed SELweb and the University’s 
IRB granted a waiver of informed consent for the research team to use de-identified SELweb and 
academic data. School personnel administered SELweb in one or two group sessions. 


SELweb data that were reasonably interpretable were shared with education partners for 
use in understanding their students and guiding instruction. 


Findings / Results 


Across the three studies, SELweb’s composite score reliabilities averaged .85 and six- 
month temporal stabilities averaged .65. In all three field trials, scores on the assessment 
modules fit a hypothesized four-factor model of Social-Emotional Comprehension. 


Overall performance on SELweb was positively associated with teacher reported social 
skill, peer acceptance, and academic competence, and negatively associated with teacher 
reported problem behavior. Standardized regression coefficients reflecting the association 
between SELweb performance and criterion measures averaged .31 (range = .21 to .46) 


Using data from the first two field trials, we tested convergent and discriminant validity 
with nested structural equation models that imposed equality constraints testing mode fit with 
convergent and discriminant paths omitted. “Convergent” paths—or paths from SELweb latent 
variables to latent variables of the same hypothesized construct using alternate measures—were 
moderate to large and significant. In addition, when each convergent path was constrained to 
zero, it significantly reduced model fit. Discriminant paths—or paths from SELweb latent 
variables to latent variables of different hypothesized SEL constructs—were small and non- 
significant. When discriminant paths were constrained to zero, model fit was not significantly 
changed. See Figure | for a summary. 


Conclusions 


Three studies provided evidence that: (a) composite assessment scores exhibited high 
reliability, (b) together, assessment modules demonstrated a theoretically coherent factor 
structure, (c) factor scores demonstrated convergent and discriminant validity, and (d) a score 
reflecting overall SELweb performance demonstrated good evidence of criterion-related validity. 
In addition, SELweb is highly usable, easily scalable, and feasibly administered in school 
settings. Educators find SELweb informative, and further work is needed to ensure that SELweb 
assessment findings guide educators to use evidence-based instructional practices. 


Demonstration 


The widespread use and increasing appetite for SELweb stems as much or more from its 
design features as from its psychometric properties. Alongside a traditional research poster 
presenting empirical findings, we will host an interactive display highlighting design features 
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that facilitated SELweb’s broad adoption and enthusiastic reception among education partners, 
including: (a) ease of use; (b) self-administering features such as voice-over and illustration; and 
(c) automated score reporting. Those who come to the poster will be able to sample from these 
features using headphones and a laptop computer. See Figure 2. 
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Appendix B. Tables and Figures 


Table | 
Description of SELweb modules, questions, and item scoring 
Module Stimulus Question and Response Options Item Score 
Social Respondents view individual = What is the child feeling? 2 Correctly recognizes emotion; | 
Awareness child faces and indicate Happy, Sad, Angry, Scared, Just OK Mistakes emotion for neutral; 0 
emotion expressed. Selects incorrect emotion 
Social Respondents hear illustrated, | Questions about character intention 2 Correct mental state inference (e 
Meaning narrated vignette and must (e.g., “Why did the boy look inthe “He thinks it is in the basket.”) 
infer the mental state of a basket?”) 1 Correct answer, no mental state 
character. Example: A boy has jjystrated, narrated forced choice, _ inference (e.g. “He looks in the 
a false belief about the location foyr possible responses. basket.”) 
of a soccer ball and looks in the pao 
Sianeplace 0 Incorrect answer (e.g. “His 
; brother told him to look there.”’) 
Social Respondents hear illustrated, Problem Identification (Study 1) 2 Descriptive (e.g. “Someone 
Reasoning narrated vignettes involving What is the problem? bumped into you.) 
ae eee Leaks Illustrated, narrated forced choice 1 Resilient (e.g. “There is no 
emaeures P y (e.g. “There is no problem”; problem”) 
classmate) or peer entry(e.g.  ¢ oe ; s 
Give jin aonvolie aa Someone bumped you”; “You feel 0 Reactive (e.g. “Someone 
bad”; “Someone bumped you & you bumped into you & you feel 
of basketball). - 3 
feel bad’’) bad.””) 
Attribution (Study 2) 2 “no” 
Did the person do it to be mean? “yes” and “a little” 
Yes or no; if yes, a little or a lot? O“yes” and “a lot” 
Goal Preference Study 1 
How do you want it to turn out? 1Positive goal; 0 Negative goal; 
Narrated forced choice with positive Study 2 
(e.g. “Become friends”) or 2 Positive goal; 1 Retribution 
retribution (e.g. “Get back at them.”) goal; 0 Revenge goal 
options 
Solution Preference 2 Competent assertive (e.g. “Talk 
What would you do? to him”); 1 Self-advocacy (e.g. 
Illustrated, narrated forced choice, sk ee teaches for help ) ane 
ete ignoring (e.g. “Walk away”); 0 
four response types (e.g. “Hit or yell Aggressive (e.g. “Hit him.”) 
at him,”; “Ask the teacher for help”; ger a : 
“Talk to him”; and “Walk away.”) 
Self- Children send illustrated rocket Children are told to get as many 3 Slowest rocket; 2 Medium 
Control: ships to space. One is fast. One points as possible in ten trials. rocket; 3 Fast rocket 
Choice is slower. One is very slow. 
Delay Task 
Self- Children view pairs of shapes Children click on a “vif the shapes 1 Correct response; 0 Incorrect 
Control: and indicate whether they are the same and an “X” if they are response 


Frustration match. Several items are 
Tolerance programmed to get “stuck.” 


different. Children do as many items 


as possible in 90 sec. 
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Figure 1 Convergent and discriminant validity 
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Figure 2 Sample interaction demonstration screen shots 
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Select a Usability Feature 
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